Papers with evaluation measure
Quantified Reproducibility Assessment of NLP Results (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for reproducibility assessment are based on concepts and definitions from metrology. |
| Approach: | They propose a method for quantified reproducibility assessment that is based on metrology. |
| Outcome: | The proposed method produces comparable scores across multiple studies . authors find that it facilitates insights into causes of variation between studies - and conclusions can be drawn about improvements. |
Evaluating Dynamic Topic Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation measures to evaluate the progression of topics in dynamic topic models (DTMs) are difficult due to their unsupervised nature, but are crucial for detecting trends in time-indexed documents. |
| Approach: | They propose to combine topic quality and temporal consistency to evaluate the progression of topics over time in dynamic topic models. |
| Outcome: | The proposed measure correlates well with human judgment and can be used to identify changing topics and evaluate different models and LLMs. |
Preventing Critical Scoring Errors in Short Answer Scoring with Confidence Estimation (2020.acl-srw)
Copied to clipboard
Hiroaki Funayama, Shota Sasaki, Yuichiroh Matsubayashi, Tomoya Mizumoto, Jun Suzuki, Masato Mita, Kentaro Inui
| Challenge: | Recent Short Answer Scoring systems use Quadratic Weighted Kappa (QWK) but it is unsatisfactory when measuring their effectiveness in actual usage. |
| Approach: | They propose a task formulation of Short Answer Scoring (SAS) that matches actual usage and extracts as many scoring predictions that are not critical scoring errors (CSEs). |
| Outcome: | The proposed system predicts scores with zero critical scoring errors (CSEs) for 50% of test data at maximum by filtering out low-reliability predictions on the basis of a certain confidence estimation. |
Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite (D19-1)
Copied to clipboard
| Challenge: | Currently, machine translation is performed at the level of individual sentences, in isolation from the rest of the document. |
| Approach: | They propose a dataset that can be used as a test suite for pronoun translation . they propose an evaluation measure to differentiate good and bad pronounce translations . |
| Outcome: | The proposed dataset can be used as a test suite for pronoun translation in English . it covers multiple source languages and different pronouner errors drawn from real system translations . |
Boosting Code Summarization by Embedding Code Structures (2022.coling-1)
Copied to clipboard
| Challenge: | Recent work on code summarization relies on structural information from the abstract syntax tree (AST) of source codes. |
| Approach: | They propose a program dependency graph (PDG) that represents the structure of a code more effectively. |
| Outcome: | The proposed model improves the performance of an out-of-domain benchmark dataset and the measure SBERT score. |
Fine-Grained Analysis of Propaganda in News Article (D19-1)
Copied to clipboard
| Challenge: | Existing methods for detecting propaganda are noisy and lack of explainability. |
| Approach: | They propose to perform fine-grained analysis of texts by detecting all fragments that contain propaganda techniques as well as their type. |
| Outcome: | The proposed model outperforms several strong BERT-based baselines. |